{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "# TensorFlow 애드온 레이어: WeightNormalization\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td><a target=\"_blank\" href=\"https://www.tensorflow.org/addons/tutorials/layers_weightnormalization\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">TensorFlow.org에서 보기</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ko/addons/tutorials/layers_weightnormalization.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Google Colab에서 실행하기</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ko/addons/tutorials/layers_weightnormalization.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">GitHub에서 소스 보기</a></td>\n",
        "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ko/addons/tutorials/layers_weightnormalization.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">노트북 다운로드하기</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "## 개요\n",
        "\n",
        "이 노트북은 가중치 정규화 레이어를 사용하는 방법과 수렴을 향상할 수 있는 방법을 보여줍니다.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KR01t9v_fxbT"
      },
      "source": [
        "# WeightNormalization\n",
        "\n",
        "심층 신경망의 훈련을 가속하기 위한 간단한 재매개변수화:\n",
        "\n",
        "Tim Salimans, Diederik P. Kingma (2016)\n",
        "\n",
        "> 이러한 방식으로 가중치를 재매개변수화함으로써 최적화 문제의 처리를 개선하고 확률적 경사 하강의 수렴을 가속합니다. 재매개변수화는 배치 정규화에서 영감을 얻었지만, 미니 배치의 예제 간에 종속성을 도입하지는 않습니다. 이는 이 방법이 배치 정규화가 덜 적합한 LSTM과 같은 반복 모델과 심층 강화 학습 또는 생성 모델과 같은 노이즈에 민감한 애플리케이션에 성공적으로 적용될 수 있음을 의미합니다. 이 방법은 훨씬 간단하지만, 전체 배치 정규화의 속도를 크게 향상합니다. 또한, 이 방법의 계산 오버헤드가 더 적으므로 같은 시간에 더 많은 최적화 단계를 수행할 수 있습니다.\n",
        "\n",
        "> https://arxiv.org/abs/1602.07868\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/seanpmorgan/tf-weightnorm/master/static/wrapped-graph.png\" width=\"80%\"><br><br>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MUXex9ctTuDB"
      },
      "source": [
        "## 설정"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CyWHXw9mQ6mp"
      },
      "outputs": [],
      "source": [
        "!pip install -U tensorflow-addons"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OywLbs7EXiE_"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_addons as tfa"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KQMhhq1qXiFF"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from matplotlib import pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ULWHqMAnTVZD"
      },
      "outputs": [],
      "source": [
        "# Hyper Parameters\n",
        "batch_size = 32\n",
        "epochs = 10\n",
        "num_classes=10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fhM0ieDpSnKh"
      },
      "source": [
        "## 모델 빌드하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7XZXnBYgRPSk"
      },
      "outputs": [],
      "source": [
        "# Standard ConvNet\n",
        "reg_model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Conv2D(6, 5, activation='relu'),\n",
        "    tf.keras.layers.MaxPooling2D(2, 2),\n",
        "    tf.keras.layers.Conv2D(16, 5, activation='relu'),\n",
        "    tf.keras.layers.MaxPooling2D(2, 2),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dense(120, activation='relu'),\n",
        "    tf.keras.layers.Dense(84, activation='relu'),\n",
        "    tf.keras.layers.Dense(num_classes, activation='softmax'),\n",
        "])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UZd6V90eR4Gm"
      },
      "outputs": [],
      "source": [
        "# WeightNorm ConvNet\n",
        "wn_model = tf.keras.Sequential([\n",
        "    tfa.layers.WeightNormalization(tf.keras.layers.Conv2D(6, 5, activation='relu')),\n",
        "    tf.keras.layers.MaxPooling2D(2, 2),\n",
        "    tfa.layers.WeightNormalization(tf.keras.layers.Conv2D(16, 5, activation='relu')),\n",
        "    tf.keras.layers.MaxPooling2D(2, 2),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tfa.layers.WeightNormalization(tf.keras.layers.Dense(120, activation='relu')),\n",
        "    tfa.layers.WeightNormalization(tf.keras.layers.Dense(84, activation='relu')),\n",
        "    tfa.layers.WeightNormalization(tf.keras.layers.Dense(num_classes, activation='softmax')),\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AA5dti8AS2Y7"
      },
      "source": [
        "## 데이터 로드하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L8Isjc7W8MEn"
      },
      "outputs": [],
      "source": [
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()\n",
        "\n",
        "# Convert class vectors to binary class matrices.\n",
        "y_train = tf.keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = tf.keras.utils.to_categorical(y_test, num_classes)\n",
        "\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cH1CG9E7S34C"
      },
      "source": [
        "## 모델 훈련하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EvNKxfaI7vSm"
      },
      "outputs": [],
      "source": [
        "reg_model.compile(optimizer='adam', \n",
        "                  loss='categorical_crossentropy',\n",
        "                  metrics=['accuracy'])\n",
        "\n",
        "reg_history = reg_model.fit(x_train, y_train,\n",
        "                            batch_size=batch_size,\n",
        "                            epochs=epochs,\n",
        "                            validation_data=(x_test, y_test),\n",
        "                            shuffle=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "esmMh-5g7wmp"
      },
      "outputs": [],
      "source": [
        "wn_model.compile(optimizer='adam', \n",
        "                 loss='categorical_crossentropy',\n",
        "                 metrics=['accuracy'])\n",
        "\n",
        "wn_history = wn_model.fit(x_train, y_train,\n",
        "                          batch_size=batch_size,\n",
        "                          epochs=epochs,\n",
        "                          validation_data=(x_test, y_test),\n",
        "                          shuffle=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yujf2YRbwX55"
      },
      "outputs": [],
      "source": [
        "reg_accuracy = reg_history.history['accuracy']\n",
        "wn_accuracy = wn_history.history['accuracy']\n",
        "\n",
        "plt.plot(np.linspace(0, epochs,  epochs), reg_accuracy,\n",
        "             color='red', label='Regular ConvNet')\n",
        "\n",
        "plt.plot(np.linspace(0, epochs, epochs), wn_accuracy,\n",
        "         color='blue', label='WeightNorm ConvNet')\n",
        "\n",
        "plt.title('WeightNorm Accuracy Comparison')\n",
        "plt.legend()\n",
        "plt.grid(True)\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "layers_weightnormalization.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
